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PREFACE 


In  June  1984,  the  ADTAC  Standardi z at i an/Eval uat i on  team 
concluded  the  Formal  Visit  for  a  unit  that  came  extremely  close 
to  earning  an  overall  Outstanding  rating  for  the 
visit — something  the  team  members  had  never  seen.  In  fact,  the 
unit  was  so  close  that  the  programs  portion  of  the  visit  was 
rated  Outstanding.  It  was  later  remarked  by  a  member  of  the 
unit  leadership  that  the  rating  was  a  result  of  a  concentrated 
effort  to  reaccomplish  every  AF  Fo*-m  8,  every  Operating 
Instruction,  every  piece  of  paper  generated  by  the  Stan/Eval 
function  that  could  possibly  be  of  interest  to  the  headquarters 
SEFEs.  Upon  further  examination,  this  type  of  preparation  for 
the  programs  portion  of  the  Formal  Visit  is  not  unusual.  A 
trend  of  this  type  not  only  degrades  the  primary  intent  of 
Stan/Eval  programs,  it  dilutes  the  purpose  and  value  of  the 
Formal  Visit.  This  paper  examines  the  higher  headquarters 
Stan/Eval  evaluation  process  from  both  an  Inspector  General 
functional  staff  perspective.  It  attempts  to  define  an 
evaluation  process  which  will  restore  the  proper  emphasis  on 
administrative  details  and  encourage  units  to  devote  primary 
emphasis  on  the  operational  performance  factors  and  combat 
capability. 
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EXECUTIVE  SUMMARY 


insights  into  tomorrow” 


Part  of  our  College  mission  is  distribution  of  the 
students’  problem  solving  products  to  DoD 
sponsors  and  other  interested  agencies  to 
enhance  insight  into  contemporary,  defense 
related  issues.  While  the  College  has  accepted  this 
product  as  meeting  academic  requirements  for 
graduation,  the  views  and  opinions  expressed  or 
implied  are  solely  those  of  the  author  and  should 
not  be  construed  as  carrying  official  sanction. 


REPORT  NUMBER  86-1885 
AUTHOR(S)  MAJOR  MICHAEL  D.  NOTLEY,  USAF 

TITLE  RECOMMENDED  CHANGES  TO  TAC  STAN/EVAL  FORMAL  VISIT 
POLICIES  AND  PROCEDURES 

Purpose;  The  Stan/Eval  -function  is  one  o-f  the  most  -frequently 
evaluated  areas  within  the  unit.  It  receives  scrutiny  during  both 
the  MEI  and  Stan/Eval  Formal  Visit  plus  other  directed  sta-f-f 
assistance  visits  and  any  required  sel -f -assessment .  The  purpose 
o-f  this  -ibajper  is  to  determine  i-f  both  the  MEI  and  the  Formal 
Visit  evaluations  are  required  to  accurately  assess  the 
-function's  capability  to  -fulfill  its  charter. 

Prob 1 em;  Three  different  approaches  were  used  to  compare  the  MEI 
and  the  Formal  Visit  in  order  to  establish  areas  of  commonality. 
First,  a  search  of  the  appropriate  directives  was  conducted  to 
determine  the  frequency,  purpose,  and  requirements  of  each  type 
evaluation.  Also,  the  contention  that  the  Formal  Visit  is  a  more 
thorough  evaluation  of  the  Stan/Eval  function  than  the  MEI  was 
studied,  and  the  rating  system  used  bv  each  type  evaluation  was 
compared.  The  second  approach  involved  a  mathematical  analysis  of 
the  MEI  and  Formal  Visit  Stan/Eval  programs  ratings  for  1984  and 
the  first  half  of  1985,  and  probability  functions  were  derived 
showing  each  evaluation  team’s  rating  tendencies.  This  was  done 
to  show  the  effect  of  differing  evaluation  philosophies  on  the 
ratings  of  various  units’  Stan/Eval  function.  Finally,  a  survey 
of  randomly  chosen  Chiefs  of  Stan/Eval  was  conducted  to  determine 
how  they  prepared  for  past  evaluations. 


CONTINUED 


Concl usions:  The  Stan/Eval  programs  portion  of  the  Formal  Visit 
and  the  MEI  o-f  the  Stan/Eval  -function  compare  quite  -favorably. 
They  evaluate  the  same  areas,  use  the  same  checklist,  and 
essentially  have  the  same  purpose  o-f  evaluating  the  management 
aspects  o-f  the  Stan/Eval  area.  Probability  -functions  that 
describe  the  rating  tendencies  of  the  evaluation  teams  are  nearly 
coincidental:  and,  within  limits,  both  the  MEI  and  the  Formal 
Visit  award  the  same  ratings  when  evaluating  similar  situations. 
Although  the  argument  supporting  the  thoroughness  of  the  Formal 
Visit  has  validity,  the  ability  of  the  MEI  to  accurately  assess 
the  function's  true  rating  supports  the  argument  that  the  IG's 
sampling  approach  is  good  enough.  One  area  of  the  Formal  Visit, 
aircrew  performance,  has  no  close  counterpart  in  the  MEI,  and 
retains  its  value  because  of  the  unique  way  it  measures  a  unit's 
ability  to  perform  mission  tasking. 


Survey  results  indicate  units  prepare  for  the  MEI  and  the  Formal 
Visit  in  much  the  same  way.  The  preponderance  of  preparation 
efforts  are  centered  in  the  Stan/Eval  programs  or  "paperwork” 
portion.  A  significant  number  of  units  go  to  great  lengths  to 
ensure  the  programs  are  inspection  ready.  Conversely,  relatively 
little  effort  is  spent  preparing  for  the  aircew  performance 
portion  which  constitutes  the  "bigger  half"  of  the  Formal  Visit 
overall  rating.  This  misplaced  effort  is  the  result  of 
overemphasis  on  the  Stan/Eval  programs  caused  by  HQ  TAC  requiring 
double  inspection  of  the  function. 


Recommenda ti  ons : 

1)  Eliminate  the  Stan/Eval  programs  as  evaluation 
the  Formal  Visit. 


items  during 


2)  Expand  the  aircrew  performance  portion  of  the  Formal  Visit 
require  evaluations  for  each  SEFE  and  instructor  within  the  un 
and  base  the  overall  unit  rating  solely  on  aircrew  performance. 


3)  Informal  visits  by  headquarters  SEFEs  should  include  spot 
checks  of  unit  programs  to  ensure  quality  remains  high. 


4)  Leave  the  MEI  unchanged. 
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Chapter  One 


C I'j 

INTRODUCTION  ‘  r  *  ... 

The  Standardization/Evaluation (Stan/Eval >  -function  in  the 
flying  unit  is  chartered  with  providing  "commanders  with 
meaningful  indicators  to  show  the  effectiveness  of  training 
that  pertains  to  command  mission  and  aircrew  flying  duties." 
af9r4i  >  Tactical  Air  Command  has  further  refined  this  charter 
"to  provide  commanders  with  meaningful  indicators  which  reflect 
aircrew  training  3nd  aircrew  ability  to  perform  the  unit 
mission. (5:  2-1  >  The  key  word  in  both  charter  statements  is 
training.  W<thin  TAC,  training,  in  one  of  its  several  forms,  is 
a  continual  pXpcess.  Those  forms  include  initial  qualification 
training,  mi ssi orr  qual i f i cat i on  training,  and  continuation 
training.  (4:  l-3)i*Stan/Eval '  s  function  is  to  measure  the 
quality  of  output  of  each  phase  of  training.  Through  the 
measuring  devices  of  individual  aircrew  evaluations  and 
collective  trend  analysis  programs,^  Stan/Eval  provides  the 
commander  the  assessment  he  needs>6  determine  aircrew 
capability  to  perform  the  un4-t-''missi  on  and,  hence,  the 
effectiveness  af  the  respective  training  program. 
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role  Stan/Eval  plays  within  the  unit  is  reflects 
iny  the  function  receives  during  formal 
It  is  inspected  during  the  Management 
ss  Inspecticns^MEI >  conducted  by  the  Inspector 
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the  results 
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N-XJaefuncti on  is  also  evaluated  during  Stan/Eval 
ts  cori3trrtsxr-br--t-he__cssponsi  bl  e  headquarters' 
earn.  It  receives  f urther~he’adqxt«r-,teci_  scrut i ny 
rmal  visits  and  staff  assistance  visits»The 
ovides  a  continual  self-assessment  through  the 
of  flight  examiner  <SEFE>  objectivity  by  the  Chief 
1  and  the  use  of  sel f -i nspect i on  checkl i sts^/ In 
he  Stan/Eval  Review  Board  meets  rout  i  nqi  y-  to  analyze 
of  unit  efforts— as- measured  ~by"~the  Stan'Eval 
n  al 1 ,  Stan/Eval  is  one  of  the  most  critically 
unctions  within  the  unit. 
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•The  purpose  of  this  is  to  determine  if  both  ME I 

Formal  Visit  evaluations  are  required  to  accurately  assess 
Stan/Eval  function's  capability  to  measure  the  quality  of 
output  of  the  unit  training  progr ams .yJo  begin.  Chapter  2 
presents  a  comparison  of  MEI  and  Formal  Visit  evaluation 
frequencies,  goals,  and  evaluation  requirements  to  determi 


and 

the 


m 


areas  of  commonality  and  divergence.  Chapter  3  will  include  the 
actual  results  -for  both  MEI  and  Formal  Visit  evaluations  -for 
the  year  and  a  half  period  beginning  in  January  1984.  Chapter  4 
documents  the  results  of  interviews  with  unit  Chiefs  of 
Stan/Eval  conducted  to  support  the  purpose  statement.  Finally, 
Chapter  5  will  conclude  with  a  summary  of  lessons  learned  and 
recommendati ons  for  future  evaluations  of  the  Stan/Eval 
function . 


Chapter  Two 


A  COMPARISON  OF  THE  ME I  AND  THE  FORMAL  VISIT 

Before  starting  a  comparative. analysis  of  the  MEI  and  the 
Formal  Visit,  it  would  be  appropriate  to  discuss  the  nature  of 
each  evaluation.  The  MEI  is  called  an  inspection  because  it  is 
administered  by  the  Inspector  General.  The  Formal  Visit  cannot 
be  called  an  inspection  because  it  is  administered  by  the 
Stan/Eval  Division,  a  subfunction  of  the  Operations  Directorate 
at  the  MAJCOM  and  Numbered  Air  Force  level.  The  MEI  evaluates 
each  functional  area  within  a  flying  unit,  including  Stan/Eval, 
and  combines  those  ratings  to  derive  an  overall  unit 
effectiveness  rating.  The  Formal  Visit  presumably  evaluates 
only  the  Stan/Eval  function  within  the  unit.  Indirectly, 
however,  other  functional  areas  are  evaluated,  especially 
during  the  aircrew  performance  portion  of  the  visit.  Two  areas, 
aircrew  training  and  weapons  and  tactics,  often  have  a  major 
impact  on  the  overall  unit  rating.  In  at  least  one  case,  a  poor 
showing  by  the  unit  training  function  during  a  Formal  Visit 
contributed  to  an  overall  marginal  rating  and  a  subsequent 
Stan/Eval  Revi si t . ( 10: 0)  Regardless  of  whether  they're  called 
inspections  or  visits,  both  are  major  evaluations  for  the  unit. 
The  results  of  each  are  briefed  at  the  command  level,  and 
extensive  reports  are  published  for  command  viewing. 


EVALUATION  FREQUENCY 


The  first  step  in  the  comparison  process  concerns  the 
frequency  of  evaluation  for  the  MEI  and  the  Formal  Visit.  There 
are  two  categories  of  units  to  be  considered:  TAC  units  and 
TAC-gained  units.  TAC-gained  units  include  Air  Reserve 
Farce (ARF)  units  of  the  Air  National  Guard  and  Air  Farce 
Reserve.  For  TAC  units  who  do  not  receive  an  Operational 
Readiness  Inspecti on (ORI )  such  as  a  Tactical  Training  Wing,  the 
MEI  occurs  each  IB  months.  All  other  TAC  units  receive  an  MEI 
each  24  months.  TAC-gained  units  can  expect  an  MEI  every  30-36 
months. (7:2)  On  the  other  hand,  the  Formal  Visit  occurs  each  20 
months  for  all  TAC  units,  training  or  otherwise. (5:2-2)  For 
TAC-gained  units,  a  recent  change  to  TACR  60-2,  Vol  I,  allows 
the  Formal  Visit  to  run  concurrent  with  the  MEI  each  36  months. 
The  typical  TAC  unit  can  expect  a  major  evaluation  every  10-12 
months,  not  including  the  ORI  if  applicable.  Although  the 
evaluation  frequency  for  the  TAC-gained  units  has  recently  been 


extended,  the  rationale  -for  the  change  may  equally  apply  to  all 
TAC  units. 


EVALUATION  PURPOSE 

The  next  step  in  the  comparison  process  involves  the 
purpose  -for  each  of  the  two  evaluations.  The  MEI  "should 
evaluate  each  functional  area  for  its  capability  to  accomplish 
assigned  responsibilities  and  tasks. “ (8: 12-1 )  It  does  this  by 
focusing  "on  economical  management  of  resources,  unit 
management  processes,  compliance  with  current  applicable 
directives,  and  MAJCOM  designated  special  inspection 
i terns. " (9: 12-1 )  The  congruency  in  these  purpose  statements  is 
the  emphasis  on  the  management  aspects  of  the  functional  area 
to  be  inspected,  in  this  case  the  Stan/Eval  function.  The 
Formal  Visit,  on  the  other  hand,  stresses  two  main  areas:  the 
Stan/Eval  program  and  the  unit  aircrew  performance.  Formal 
Visit  goals,  listed  in  TACF:  60-2,  Vol  I,  include: 

1)  Determine  the  effectiveness  of  the  Stan/Eval 
programs. 

2)  Assess  the  capability  of  the  Chief  of  Stan/Eval, 
flight  examiners,  instructors,  and  unit  aircrew 
members  through  flight  evaluations  and  written 
evaluations  to  evaluate,  instruct  and  perform  the 
unit’s  assigned  tasking. 

3)  Review  unit  Stan/Eval  administrative  procedures 
and  records  to  include  all  publications  and  written 
directives  pertaining  to  flying  operations. 

4)  Identify  operational  or  training  factors  adversely 
affecting  aircrew  capability  to  accomplish  assigned 
mission  and  recommend  corrective  action  as  required. 

5)  Verify  aircrew  compliance  with  approved 
operational  procedures. 

It  is  apparent  from  Objectives  1,3,4,  and  5  that  the  Formal 
Visit  has  a  great  deal  in  common  with  the  MEI,  at  least  in 
purpose.  All  four  goals  stress  the  management  of  the  Stan/Eval 
function.  However,  arguments  can  be  made  that  the  Formal  Visit 
best  determines  the  management  effectiveness  and  quality  of  the 
administrative  portion  of  the  Stan/Eval  function.  This  argument 
will  be  addressed  following  a  comparison  of  evaluation 
requi rements. 


AIRCREW  PERFORMANCE 

Before  conducting  a  comparison  of  evaluation  requirements, 
a  main  area  o-f  the  Formal  Visit  not  yet  considered  is  the 
aircrew  performance  portion  of  the  evaluation.  A  major  portion 
of  the  Formal  Visit  consists  of  measuring  the  quality  of 
aircrew  performance  during  flight  evaluations,  emergency 
procedures  evaluations,  and  written  examinations.  Performance 
standards  and  evaluation  criteria  are  detailed  by  regulation. 
The  overal 1  rating  for  aircrew  performance  is  a  subjective 
summation  of  the  unit’s  performance  of  objective  tasks  and  is  a 
significant  factor  in  the  unit’s  overall  Formal  Visit  rating. 
Likewise,  the  MEI  measures  aircrew  performance  but  in  a  more 
indirect  manner.  Although  no  flight  evaluations (check  rides) 
are  flown,  mission  evaluations  are.  Regulation  directs  that 
only  Stan/Eval  flight  ex  ami ner s ( SEFEs )  may  conduct  flight 
evaluations,  and  generally  speaking,  there  are  no  SEFEs  on  the 
IG  team. (5:1-1)  Flight  evaluations  differ  from  mission 
evaluations  in  both  scope  and  depth.  The  mission  evaluation  is 
used  to  assess  the  appropr i ateness  and  adequacy  of  unit 
training  missions  in  preparing  aircrews  for  combat  tasks.  A 
necessary  fallout  from  this  type  evaluation  is  an  assessment  of 
the  aircew  member's  ability  to  safely  conduct  training.  This 
assessment  is  based  more  on  the  inspector’s  overall  impression 
rather  than  on  any  specific  mission  event.  Flight  evaluations, 
on  the  other  hand,  are  task  oriented,  and  objective  performance 
criteria  is  listed  for  each  event  the  aircrew  might  perform.  As 
an  example,  a  flight  evaluation  would  include  specific  events 
such  as  takeoff,  navigation,  instrument  approach,  weapons 
employment,  etc.,  and  assign  them  individual  grades  according 
to  established  cri teria. (6: 5-1 )  Conversely,  the  mission 
evaluation  might  look  at  the  same  events,  and  yet  the  only 
criteria  it  has  to  meet  is,  "Does  the  aircraft  commander 
maintain  positive  control  through  all  phases  of  f 1 i ght?" ( 2: 1 ) 

The  Formal  Visit  will  also  fly  flight  evaluations  on  a 
higher  percentage  of  unit  aircrew  members.  First  Air  Force 
(previously  ADTAC)  Stan/Eval  used  a  goal  of  40  percent  of 
available  aircrews.  The  typical  MEI  would  only  fly  a  third  as 
much.  This  is  not  to  say  that  mission  evaluations  do  not 
fulfill  IG  requirements  for  evaluating  the  management  of  unit 
flying  activities.  They  most  assuredly  do:  however,  mission 
evaluations  do  not  measure  the  quality  of  aircrew  performance 
to  the  depth  required  of  a  Formal  Visit.  Therefore,  because  the 
aircrew  performance  portion  of  the  Formal  Visit  and  the  mission 
evaluation  of  the  MEI  do  not  favorably  compare,  the  remainder 
of  this  analysis  will  deal  primarily  with  the  administrative 
portion  of  the  Stan/Eval  function,  the  Stan/Eval  program. 


EVALUATION  CHECKLISTS 


In  comparing  evaluation  requirements,  it  will  be  necessary 
to  look  both  at  the  checklist  used  by  each  type  of  evaluation 
and  how  that  checklist  is  applied.  The  checklist  requirements 
o-f  the  MEI  and  the  Formal  Visit  are  virtually  the  same.  This  is 
not  surprising  since  both  are  prepared  by  Stan/Eval  people  and 
use  the  same  regulations  as  source  material.  The  TACR  60-2 
describes  the  set-up,  operation,  and  evaluation  o-f  the  unit 
Stan/Eval  -function,  and  it  breaks  the  program  into  11  subareas. 
These  subareas  are  the  -framework  -for  both  o-f  the  evaluation 
checklists.  The  MEI  checklist,  TACR  123-42,  is  prepared  by  TAC 
Stan/Eval (HO  TAC/DQV) .  Each  Numbered  Air  Force  Stan/Eval  team 
prepares  its  own  Formal  Visit  checklist,  but  they  di-f-fer  only 
in  -format.  All  the  checklists  are  structured  the  same  with 
appropriate  evaluation  items  listed  under  each  o-f  the  11 
subareas.The  MEI  subareas  di-f-fer  -from  the  ones  listed  in  the 
Stan/Eval  checklists,  but  the  dif-ference  is  small.  It  appears 
that  this  difference  is  because  the  Formal  Visit  checklists 
have  been  updated  to  reflect  the  most  recent  changes  to  TACR 
60-2  while  TACR  123-42  has  not  been  updated  since  1?S3.  Since 
the  MEI  checklist  matches  the  old  version  of  TACR  60-2,  it  is 
assumed  that,  following  revision,  TACR  123-42  will  again  match 
the  revised  regulation.  Other  than  this  slight  difference,  the 
checklists  are  all  virtually  the  same  and  quite  interchangeable 
among  the  various  evaluation  teams. 


DEPTH  OF  EVALUATION 


While  the  checklists  may  be  the  same,  arguments  can  be 
made  that  the  primary  difference  between  the  two  evaluations 
deals  with  how  that  checklist  is  actually  applied.  Principally, 
the  Formal  Visit  provides  a  more  in-depth  look  at  the  Stan/Eval 
function  and  hence  provides  a  more  accurate  assessment.  This 
argument  is  based  on  two  suppositions.  First,  the  MEI,  because 
it  must  inspect  each  functional  area  within  the  unit,  is 
limited  in  the  time  that  can  be  devoted  to  the  Stan/Eval 
function.  Routinely,  the  Formal  Visit  devotes  a  three  to  four 
day  period  to  evaluate  the  Stan/Eval  program.  The  IE  normally 
allows  half  a  day  unless  major  problems  develop  that  require  a 
more  complete  look.  While  the  Formal  Visit  might  use  two  or 
three  evaluators  during  this  time,  the  MEI  normally  uses  only 
one. 

The  second  supposition  concludes  that  the  MEI  evaluates 
the  function  by  looking  at  the  11  subareas  as  they  appear 
during  the  evaluation.  They  take  a  "snapshot"  look.  Conversely, 
the  Formal  Visit  evaluates  those  11  subareas  by  reviewing  their 
performance  history  since  the  last  Formal  Visit.  By  way  of 
example,  one  of  the  11  subareas  in  the  Stan/Eval  program  is 
titled  "Flight  Evaluation  Fol ders (FEF) /  Preparation  and  Control 


of  AF  Forms  8."  (5:2-3)  The  requirement  -for  FEFs  and  AF  Forms  8 
is  established  by  AFF:  60-1.  These  items  are  permanent  documents 
maintained  on  each  aircrew  member  describing  their 
qual  i  f  i  cati  on  history  and  performance  on  required  flight 
evaluations.  Command  documentation  procedures  are  listed  in 
TACF:  60-2  and  are  included  as  evaluation  items  on  both  the  MEI 
and  Formal  Visit  checklists.  Evaluators  for  the  Formal  Visit 
will  generally  look  at  every  FEF  and  every  AF  Form  8  generated 
since  the  last  Formal  Visit.  The  MEI  generally  only  looks  at  a 
small  sample  of  these  items. 

Obviously,  both  suppositions  have  merit.  More  inspectors 
devoting  more  time  looking  at  the  total  sum  of  the  Stan/Eval 
function  should  provide  a  more  in-depth  look  than  the  MEI  can 
provide.  But  the  question  hinges  on  whether  the  MEI  look  is 
good  enough  to  make  an  accurate  assessment  of  the  program. 
Interviews  with  IG  personnel  indicate  they  feel  the  depth  of 
evaluation  is  sufficient,  and  empirical  data  will  be  introduced 
in  the  next  chapter  to  document  the  accuracy  of  the  MEI 
assessment. 


THE  RATING  SYSTEM 

The  final  area  that  requires  comparison  is  the  rating 
system  used  during  the  two  evaluations.  Both  have  five  tier 
grading  systems  with  the  ratings  ranging  from  unsatisfactory  to 
outstanding.  Because  more  than  90  percent  of  Stan/Eval  function 
ratings  fall  into  the  satisfactory  to  excellent  range,  this 
analysis  will  concentrate  on  those  rating  definitions.  Key 
phrases  of  each  definition  are  included  below. 

MEI  RATING  DEFINITIONS 

Excellent-  907.  of  HQ  TAC  inspection  guide  items  must 
be  accomplished  in  a  superior  manner.  Management 
of  ...  programs ... rel at i vel y  free  of 
deficiencies.  <8: 1 2— 1 > 

Satisfactory-  75V.  of  HQ  TAC  inspection  guide  items 
accomplished  in  a  competent  manner.  Management ... i s 
efficient.  Minor  deficiencies  may  exist,  but  do  not 
impede  or  limit  mission  accomp 1 i shment . ( 8 : 1 2- 1 ) 

FORMAL  VISIT  DEFINITIONS 

Excellent-  Procedures  in  effect  exceeded  requirements 
and  enhanced  overall  ef f ecti veness. (5: 2-5) 

Sati sf actory-  Perf ormance. . . meets  mission 
requirements.  ...programs  are  efficiently  managed. 

Minor  deficiencies  may  exist,  however  they  do  not 
impede  or  limit  mission  accompl i shment . (5: 2-5) 


The  percentage  -factors  given  in  the  MEI  de-finitions  stick 
out  as  the  main  difference  between  the  two  rating  systems. 

Those  percentages  serve  only  as  a  guide.  The  judgement  of  the 
inspector  is  the  overriding  f actor . (8: 12— 1 )  Regardless,  the 
satisfactory  rating  definitions  display  enough  similarities  to 
be  consistent.  The  excellent  ratings  diverge  somewhat, 
especially  in  the  phrases  "relatively  free  of  deficiencies"  and 
"exceeded  requirements. "  If  a  requirement  of  the  program  is  not 
met,  then  a  deficiency  exists  by  the  Formal  Visit  definition. 

It  seems  that  an  excellent  rating  earned  through  the  Formal 
Visit  criteria  would  qualify  as  an  excellent  under  the  MEI.  The 
converse  does  not  necessarily  hold.  A  unit  achieving  only  90 
percent  of  program  requirements  could  not  be  awarded  an 
excellent  by  the  Formal  Visit  even  if  the  deficiencies  do  not 
detract  from  program  effectiveness.  The  point  of  the  discussion 
is  not  which  rating  system  is  best.  They  both  serve  the 
intended  purpose.  The  point  is  the  MEI  definition  of  excellent 
is  broader  and  should  encompass  a  higher  percentage  of  units 
when  compared  to  the  Formal  Visit.  This  fact  will  be  borne  out 
in  the  next  chapter. 


EVALUATION  SUMMARY 

Comparison  of  the  two  evaluations  has  shown  several 
things.  First,  regardless  of  the  name,  both  evaluations  are 
inspections.  Second,  the  frequency  of  these  evaluations  forces 
units  to  prepare  for  them  at  a  rate  better  than  one  per  year. 
Third,  both  evaluations  are  remarkably  similar  in  the 
evaluation  of  the  Stan/Eval  program,  although  distinct 
differences  are  apparent  in  the  aircrew  performance  portion. 
Fourth,  inspection  checklists  are  nearly  identical;  and  while 
arguments  can  be  made  that  the  Formal  Visit  provides  a  more 
in-depth  evaluation,  the  MEI  evaluation  may  prove  to  be 
sufficient.  Finally,  subtle  differencies  exist  in  the  rating 
systems  of  the  two  evaluations.  The  satisfactory  ratings  are 
consistent  between  the  two,  but  differences  in  the  excellent 
definition  actually  provide  the  MEI  a  wider  range  of 
possibilities.  The  next  chapter  will  deal  with  an  empirical 
evaluation  of  actual  MEI  and  Formal  Visit  results. 


Chapter  Three 


STAN/EVAL  PROGRAM  COMPARATIVE  ANALYSIS 

To  conduct  a  comparative  analysis,  the  actual  MEI  ratings 
-for  the  Stan/Eval  -function  and  the  Stan/Eval  program  portion  o-f 
the  Formal  Visit  ratings  were  compared.  Data  -from  CY  1984  and 
through  second  quarter  o-f  CY  1985  was  used.  The  data  is 
presented  in  tabular  form  in  figure  1. 
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Figure  1. Rating  Distribution. 


The  differences  in  the  listed  totals  are  due  to  timing 
variances  in  the  respective  evaluations.  Data  from  CY  1983  ar 
earlier  was  discarded  because  it  was  incomplete.  The  sice  of 
the  listed  sample  does  not  include  each  TAC  unit,  but  it  is 
sufficiently  large  to  offer  opportunities  for  analysis. 

To  graphically  c'  splay  this  data,  the  Gaussian  or  Normal 
distribution  function  was  chosen .( 1 : 304)  The  function, 
highlighted  in  Figure  2,  required  that  numerical  values  be 
assigned  to  the  rating  scheme  used  by  the  evaluation  teams. 
This  was  arbitrarily  chosen  to  be  unsatisf actory=l ,  marginal = 
satisf actory=3,  e:: cel  1  ent=4 ,  and  outstandi ng=5. 


This  -function,  or  the  probability  that  rating  X  will 
occur,  de-fines  the  normal  distribution  curve  that  best  -fits  the 
actual  data  -from  Figure  1.  The  mean(p)  and  the  standard 
deviation  (<r)  can  also  be  calculated  to  aid  in  the  analysis.  To 
■fit  this  data  to  a  curve,  the  least  squares  method  was 
used.  (1:134)  Although  it  is  beyond  the  scope  o-f  this  paper  to 
provide  a  mathematical  step  by  step  derivation,  the  information 
presented  in  Figure  3  was  calculated  from  the  raw  data  listed 
in  Figure  1.  The  IG  subscript  applies  to  MEI  results.  The  DOV 
subscript  applies  to  Formal  Visit  Stan/Eval  program  ratings. 


An  overlay  of  each  distribution  curve  is  included  with  a 
step  function  graph  of  the  evaluation  results  listed  in  figure 

4. 


Figure  4.  Probability  Function  Overlays. 


The  resultant  visible  difference  in  the  respective  curves 
is  due  to  the  unequal  number  of  evaluations  given.  In  order  to 
give  a  better  visual  representat i on  of  the  rating  probability 
curves,  it  was  necessary  to  normalize  the  curves.  The 
normalization  only  adjusts  the  data  to  equalize  the  sample 
size.  In  other  words,  the  shape  of  the  function  will  not 
change,  but  the  area  under  the  curves  will  be  adjusted. 
was  done  by  establishing  a  ratio  for  the  maximum  points  of  the 
two  curves.  The  ME I  data  was  arbitrarily  adjusted;  however,  e 
Formal  Visit  data  could  be  adjusted  simply  by  us*!^  1  n^:e 

of  the  normalization  factor.  That  factor  equals  6^.497  divided 

by  37.794  or  1.68. 
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Figure  5.  Graphic  Comparison. 
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point  of  each  curve.  In  this  case,  the  ranges  -for  both 
evaluations  coincide  and  include  all  possible  ratings  -from 
satisf actory  to  outstanding. 

One  last  area  requiring  examination  is  a  direct  comparison 
o-f  each  evaluation  team’s  ability  to  arrive  at  any  one  unit’s 
true  Stan/Eval  program  rating.  To  properly  compare  their  rating 
abilities,  it  would  be  necessary  to  conduct  simultaneous  MEI 
and  Formal  Visit  evaluations  o-f  the  same  Stan/Eval  -function. 
This  not  being  possible,  an  alternative  method  was  chosen.  This 
method  uses  the  previously  listed  data  but  only  for  units  that 
received  both  an  MEI  and  a  Formal  Visit  during  the  18  month 
time  period.  Data  is  included  in  figure  6. 


RATING  BREAKOUT  FOR  UNITS  RECEIVING  BOTH 
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Figure  6.  Rating  Distribution. 


Although  the  data  appears  to  be  weighted  more  towards 
excellent  for  the  MEI,  it  is  a  reasonable  fit  to  the 
probability  functions  listed  earlier.  It  would  be  convenient  if 
the  ratings  matched  one  for  one  between  the  evaluation  teams. 
This  would  greatly  simplify  the  task  of  showing  rating 
consistency  between  the  teams.  Such  is  not  the  case.  Of  the  51 
units,  only  27  received  the  same  rating  on  the  MEI  as  on  the 
Formal  Visit  Stan/Eval  program  rating.  But  because  of  the 
factors  that  cannot  be  held  constant,  the  statistical  validity 
of  this  simple  ratio  as  an  indicator  of  relative  rating  ability 
is  questionable.  Factors  such  as  the  learning  process  that 
takes  place  between  evaluations  or  personnel  changes  within  the 
unit  command  structure  or  Stan/Eval  function  itself  can  affect 
the  rating.  Theoretically,  the  more  closely  spaced  the 
evaluations  are,  the  greater  likelihood  of  them  seeing  the 
"same"  program.  However,  twelve  units  each  had  the  MEI  and  the 
Formal  Visit  within  a  three  month  time  frame.  Seven  of  the  12 


improved  by  one  rating  -from  one  evaluation  to  the  next.  Is  this 
a  rating  inconsistency  between  the  two  evaluation  teams  or  the 
result  of  the  learning  process  that  takes  place  -following  an 
evaluation?  One  would  think  the  latter  choice  is  the  most 
likely  case?  but,  statistically,  it  can't  be  proven.  Therefore, 
a  new  test  was  needed  to  determine  rating  consistency  between 
the  two  teams. 

The  test  chosen  to  determine  consistency  includes  two 
criteria.  First,  the  ratings  -for  any  particular  unit  should 
■fall  within  some  range  of  each  other.  In  other  words,  it  would 
be  unlikely  i-f  there  were  a  drastic  change  in  the  program  from 
one  evaluation  to  the  next.  The  range  chosen  for  this  criteria 
was  +  or  -  one  rating.  Statistically,  that  is  a  rather  large 
range,  but  it  is  the  smallest  measurable  unit  within  the 
present  rating  system.  The  second  criteria  establishes  that  in 
no  case  would  one  evaluation  team  rate  a  unit  satisfactory  or 
better  and  the  next  evaluation  team  find  the  unit  less  than 
satisf actory.  It  would  be  questionable  if  one  evaluation  team 
rated  a  unit  Stan/Eval  program  satisfactory  and  three  months 
later  the  other  team  found  the  same  unit  marginal  or  worse. 
Although  these  criteria  may  not  constitute  proof  that  the  teams 
are  consistent  in  their  ability  to  arrive  at  the  same  rating 
when  confronted  by  similar  situations,  these  criteria  do  offer 
a  much  more  sound  basis  for  analysis  than  the  simple 
statistical  ratio. 

Results  of  the  analysis  were  encouraging.  Of  the  51  units 
evaluated,  49  met  the  first  criteria  for  a  ratio  of  96  percent. 
The  two  exceptions  require  examination  and  will  be  discussed 
below,  but  the  high  percentage  is  an  indication  of  consistency. 
The  second  criteria  was  satisfied  even  better  as  there  were  no 
"failures."  However,  there  were  four  cases  where  one  evaluation 
team  found  the  program  marginal  and  the  other  team  would  find 
the  program  satisfactory  or  better.  Although,  it  is  reasonable 
to  expect  a  unit  to  improve  following  a  marginal  rating,  two  of 
these  four  cases  included  units  that  made  a  remarkable  recovery 
by  improving  from  a  marginal  to  an  excellent  in  only  six 
months.  These  were  the  only  exceptions  to  the  first  criteria. 
Ignoring  the  possibility  that  the  ratings  actually  reflect  the 
true  rating  for  the  unit’s  Stan/Eval  function  at  the  time  of 
the  evaluation,  any  inconsistency  between  the  teams  would  show 
up  in  these  examples.  These  examples  constitute  the  "worst 
case."  In  the  first  example,  the  Formal  Visit  found  unit  A’s 
Stan/Eval  program  marginal  in  June  1984.  By  December,  the  MEI 
found  the  program  had  improved  to  excellent.. In  the  second 
example,  the  MEI  rated  unit  B’s  Stan/Eval  function  marginal 
while  six  months  later  the  Formal  Visit  rated  the  program 
portion  excellent.  If  any  inconsistency  exists,  it  appears  the 
evaluation  teams  are  at  least  equally  inconsistent.  More 
likely,  the  ratings  are  in  fact  correct,  and  the  evaluation 
teams  are  highly  consistent  in  reaching  the  same  conclusion 


when  -faced  with  similar  situations 


SUMMARY 


In  a  comparison  o-f  MEI  ratings  with  Formal  Visit  Stan/Eval 
program  ratings,  empirical  data  shows  a  remarkable  degree  o-f 
consistency  between  the  two  teams.  Probability  functions 
calculated  from  actual  unit  ratings  show  nearly  identical 
tendencies  to  award  any  particular  rating.  Despite  limitations 
that  hinder  statistical  analysis,  the  data  also  suggests  that 
both  teams  are  equally  capable  of  measuring  the  true  rating  of 
any  particular  unit’s  Stan/Eval  program.  Taking  this 
supposition  one  step  further,  this  analysis  concludes  that 
either  evaluation  team  is  capable  of  providing  the  true  rating 
of  a  unit’s  Stan/Eval  program.  Interviews  were  conducted  with  a 
number  of  unit  Chiefs  of  Stan/Eval  who  had  recently  undergone 
the  scrutiny  of  these  evaluations.  The  results  of  those 
interviews  are  included  in  the  next  chapter. 


Chapter  Four 


INTERVIEW  SUMMARY 

In  an  effort  to  determine  typical  evaluation  preparation 
requirements  -for  the  Formal  Visit,  in-formal  telephonic 
interviews  were  conducted  with  a  cross  section  o-f  command  unit 
Chiefs  of  Stan/Eval.  Chosen  at  random  by  a  representati ve  of  HQ 
TAC/DOV,  the  sample  included  11  units  from  the  active,  Air 
National  Guard  and  Air  Force  Reserve  components.  Selection  was 
based  solely  on  results  of  the  unit's  last  Formal  Visit 
Stan/Eval  program  rating.  Units  from  each  of  the  three  Numbered 
Air  Forces  that  make  up  TAC  were  included  in  the  sample  as  was 
one  direct  reporting  unit.  Most  of  the  units  were  operational 
although  three  training  units  were  included  to  fill  the  sample 
population.  Overall,  it  was  a  fair  representati on  of  the  parent 
population  described  in  chapter  3. 

The  interview  questions  were  designed  to  determine  typical 
evaluation  preparation  requirements.  They  also  allowed  each 
respondent  to  comment  on  any  aspect  of  the  evaluation  process. 
In  order  to  assure  complete  objectivity,  they  were  promised 
anonymity.  Overall,  every  Chief  of  Stan/Eval  was  very 
cooperative,  and  the  answers  were  quite  candid.  The  questions 
included: 

1)  What  rating  did  your  Stan/Eval  program  receive  on  the 

last  Formal  Visit? 

2)  What  did  the  ME I  rate  it? 

3)  Do  you  consider  the  Formal  Visit  an  inspection? 

4)  How  much  time  did  your  unit  devote  to  preparing  for  the 

administrative  portion  of  the  Stan/Eval  Formal  Visit? 

5)  The  aircrew  performance  portion? 

6)  Did  you  do’  anything  different  for  the  MEI? 


7)  Comments? 


VISIT?  ME I? 


The  responses  to  these  questions  are  presented  in  tabular 
•form  and  included  only  to  indicate  the  sample  taken  o-f  the 
parent  population  listed  previously. 


Formal  Visit 

Stan/Eval  Program  Rating  EESOE  ESEEES 


ME  I 

Stan/Eval  Rating 


EESOO  ESESES 


The  answers  to  this  question  supported  an  earlier 
contention  that  the  Formal  Visit  is  an  inspection  in  every  way 
but  name.  All  eleven  responses  indicate  the  Formal  Visit  is 
definitely  treated  like  an  inspection  by  the  units.  The  most 
colorful  answer  said,  "calling  it  a  visit  is  bull.  People  can 
get  fired."  However,  several  answers  indicated  the  units 
recognized  the  differences  between  the  two  evaluations.  "The 
Formal  Visit  is  more  rigorous  than  any  MEI."  It  "serves  a  good 
function"  by  keeping  the  unit  "exposed  to  a  spotlight"  and 
"keeps  aircrew  performance  up."  Another  respondent  indicated 
"the  Formal  Visit  looks  at  the  whole  unit  for  one  week  of 
flying.  The  MEI  is  general  perceptions  only.  They  both  look  at 
the  same  paperwork — sometimes  in  the  same  detail."  It  is  clear 
from  their  responses  the  units  consider  the  Formal  Visit  an 
inspection.  How  much  effort  the  unit  expends  in  preparation  for 
that  inspection  is  the  subject  of  the  next  question. 


HQUI  MUCH  TIME  DID  YOUR  UNIT  DEVOTE  TO  PREPARING  FOR  THE 
ADMINISTRATIVE  PORTION  DF  THE  STAN/EVAL  FORMAL  VISIT? 


The  responses  to  this  question  were  varied,  and  no  general 
pattern  developed  between  time  of  preparation  and  the  resultant 
program  rating  from  the  Formal  Visit.  One  unit  who  is  "75 
percent  inspection  ready  all  the  time"  spent  less  than  a  week 
preparing  for  the  evaluation.  The  Chief  of  Stan/Eval  took  one 
day  to  proofread  all  the  AF  Forms  8,  "Certificate  of  Aircrew 
Qualification,"  that  had  been  generated  since  the  last  Formal 
Visit  and  correct  a  few  minor  errors.  The  unit  was  awarded  an 
excellent  on  their  Stan/Eval  program.  Another  unit  spent  three 
months  preparing  for  the  inspection.  They  brought  in  outside 
help  and  put  in  some  70  hour  weeks  for  the  last  month.  They  had 
to  correct  roughly  90  percent  of  AF  Forms  8  generated  since  the 
last  Formal  Visit.  The  unit  was  also  rated  excellent  on  the 
Stan/Eval  program  part  of  the  inspection.  Whether  the  unit 
prepared  for  the  inspection  throughout  the  year  by  keeping  the 
program  in  inspection  order  or  made  a  mad  rush  at  the  end,  the 
rewards  appear  to  be  the  same. 

Although  there  were  varied  responses  to  the  question,  most 
units  put  in  a  significant  effort  preparing  their  Stan/Eval 
programs  just  prior  to  the  evaluation.  Typical  preparation 
times  for  these  units  ran  from  one  to  three  months  prior  to  the 
Formal  Visit.  Responses  such  as  "fairly  massive  effort," 
"rewrote  lots  of  things"  and  "brought  in  two  extra  helpers" 
were  typical.  While  it  is  impossible  to  determine  how  much  that 
effort  paid  off  in  the  form  of  better  ratings,  five  of  the 
seven  units  who  put  in  this  significant  effort  had  Stan/Eval 
programs  rated  excellent.  Since  the  first  goal  of  the  Formal 
Visit  is  to  determine  the  effectiveness  of  the  Stan/Eval 
programs,  it  would  be  interesting  to  know  if  these  five  units 
are  really  excellent  or  are  their  ratings  a  result  of  that 
"fairly  massive  effort."  Arguments  were  given  earlier 
concerning  the  depth  of  evaluation.  The  implication  was  that 
the  Formal  Visit  gave  a  more  accurate  picture  because  it  looked 
at  the  unit's  performance  history  since  the  last  Formal  Visit 
rather  than  just  taking  a  "snapshot"  such  as  the  IG  might  do. 
If,  however,  the  evaluation  teams  are  looking  at  recently 
prepared  Stan/Eval  programs,  the  depth  of  evaluation  question 
becomes  a  moot  point  because  they  are  not  necessarily  the 
actual  working  programs. 

While  these  seven  units  put  in  special  efforts  in 
preparing  the  Stan/Eval  program  for  the  Formal  Visit,  the  other 
four  did  quite  the  opposite.  Very  little  pre-inspection 
preparation  was  conducted  by  any  of  the  units.  Special  efforts 
ranged  from  "less  than  a  week"  to  "not  much."  In  fact,  one  unit 
only  had  to  "come  up  with  a  sales  pitch."  The  units  all 
received  program  ratings  of  excellent  or  better.  Why  were  they 
able  to  garner  high  ratings  without  making  a  pre-i nspecti on 


"massive  effort"?  Three  of  the  four  units  felt  the  secret  lay 
in  keeping  the  programs  functioning  all  the  time.  Inspection 
preparation  then  became  a  matter  of  only  "fine  tuning  existing 
programs. "  Putting  in  the  effort  year-round,  rather  than  a 
pre-inspection  effort,  is  obviously  the  desired  method  of 
preparing.  Not  only  is  the  pre-inspection  preparation  greatly 
simplified,  but  the  unit  gets  the  full  benefit  of  Stan/Eval 
programs  that  are,  without  a  doubt,  superior. 


HOW  MUCH  EFFORT  MAS  EXPENDED  PREPARING  FOR  THE  AIRCREW 
PERFORMANCE  PORTION  OF  THE  FORMAL  VISIT? 


Preparation  for  the  aircrew  performance  portion  of  the 
Formal  Visit  consisted  almost  entirely  in  getting  unit  aircrews 
ready  for  the  testing  portion  of  the  evaluation.  It  involved 
study  sessions  and  practice  tests  starting  three  to  six  months 
prior  to  the  Formal  Visit.  Preparation  for  the  flight 
evaluations  was  generally  "business  as  usual"  for  most  of  the 
units.  Two  units  did  try  to  fly  training  mission  profiles  that 
simulated  the  type  mission  expected  to  be  flown  during  the 
evaluation,  and  another  unit  made  special  efforts  to  improve 
mission  briefings.  The  one  unit  that  went  so  far  as  to  fly 
practice  checkride  profiles  with  the  expected  examinees  could 
only  do  so  because  it  was  a  small  unit  and  had  a  limited 
mission.  Otherwise,  preparation  for  the  aircrew  performance 
half  of  the  Formal  Visit  was  limited  to  aircrew  testing. 


DID  YOU  DO  ANYTHING  DIFFERENT  FOR  THE  ME I? 


The  units  prepared  for  the  MEI  in  much  the  same  manner 
they  prepared  for  the  Stan/Eval  program  portion  of  the  Formal 
Visit.  Although,  from  their  answers,  it  appears  the  MEI  is  less 
intense  than  the  Formal  Visit.  Several  units  commented  that 
since  the  MEI  came  on  the  heels  of  the  Formal  Visit, 
preparation  consisted  only  of  correcting  any  deficiencies  found 
during  the  Formal  Visit.  Several  also  commented  that  the  MEI 
was  less  detailed  than  the  Formal  Visit  and  they  only  needed  to 
update  the  programs  for  any  changes  since  the  inspection.  One 
unit  did  put  in  extra  effort  preparing  the  function’s  files  to 
ensure  compliance  with  the  appropriate  administrative 
directives — something  the  unit  felt  was  unnecessary  for  the 
Formal  Visit.  On  the  whole  though,  preparation  for  the  MEI  was 
the  same  as  for  the  Stan/Eval  program  portion  of  the  Formal 
Visit. 
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COMMENTS? 


Be-fore  summarizing  this  chapter,  one  area  came  up  in 
nearly  one  third  of  the  Chiefs  of  Stan/Eval  comments.  There  was 
a  perception  that  evaluation  ratings  were  based  more  on  how 
well  those  programs  were  presented  to  evaluators  than  on  their 
true  substance.  In  fact,  one  unit  felt  it  had  raised  its  rating 
by  one,  in  part,  by  doing  a  masterful  job  at  "selling"  the 
programs.  Obviously,  this  salesmanship  is  to  be  expected  as 
the  units  strive  for  the  highest  possible  ratings.  There  is  no 
evidence  to  indicate  that  salesmanship  did  any  more  than  get 
the  inspection  off  to  a  good  start.  However,  the  number  of 
units  that  had  this  impression  was  statistically  significant, 
and  evaluation  teams  should  be  reminded  that  salesmanship  will 
occur  but  should  not  affect  the  ratings. 


SUMMARY 


The  purpose  of  this  survey  was  to  build  a  picture  of  how 
the  typical  unit  Stan/Eval  function  prepares  for  a  major 
evaluation  like  the  Formal  Visit  or  MEI.  This  picture  is 
important  because  it  tells  the  evaluation  teams  how  they  affect 
a  unit  when  they  come  to  inspect.  Ideally,  the  unit’s  Stan/Eval 
function  operates  as  intended  by  the  regulation  all  the  time. 
The  Formal  Visit  would  then  only  have  to  look  at  Stan/Eval 
programs,  rate  them,  and  then  depart — having  only  minimum 
impact  on  the  unit’s  normal  training  routine.  Such  is  not  the 
case  with  most  of  the  units.  The  inspection  teams  actually  see 
specially  prepared  programs  that  have  received  much  recent 
attention.  The  units  who  were  interviewed  probably  deserved  the 
ratings  they  received.  There  is  no  evidence  to  the  contrary. 

But  the  fact  remains,  the  majority  of  the  units  put  in  a 
significant  effort  preparing  their  Stan/Eval  programs  for 
inspection,  both  by  Formal  Visit  and  MEI.  Conversely,  little 
extra  effort,  other  than  testing  preparation,  was  made  for  the 
aircrew  performance  portion  of  the  Formal  Visit.  If  aircrew 
performance  constitutes  the  "biggest  half"  of  the  Formal  Visit 
unit  rating,  it  would  appear  these  units  are  putting  their 
emphasis  in  the  wrong  place.  The  next  chapter  deals  with  this 
perception  and  offers  suggestions  on  how  it  might  be  changed. 
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Chapter  Five 


LESSONS  LEARNED  AND  RECOMMENDATIONS 

In  the  previous  chapters,  arguments  have  been  presented 
that  compare  the  Stan/Eval  program  portion  of  the  Formal  Visit 
with  the  ME  I  o-f  the  Stan/Eval  -function.  Those  arguments  have 
shown  that  the  two  evaluations  compare  quite  -favorably.  They 
both  stress  the  managerial  aspects  o-f  the  -function  and  the 
resultant  administrative  programs  and  practices.  They  use 
essentially  the  same  inspection  checklist.  Probability 
-functions  that  describe  each  inspection  team’s  tendency  to 
award  any  particular  rating  are  nearly  coincidental.  The  data 
also  implies  that  each  team,  -faced  with  the  same  situation, 
would  award  the  same  rating.  In  other  words,  both  teams  are 
equally  capable  a-f  evaluating  the  Stan/Eval  programs  portion  o-f 
the  -function  and  are  in  -fact  performing  redundant  inspections. 

When  these  two  inspections  evaluate  the  same  area,  a 
signal  is  being  sent  to  the  unit  that  the  area  must,  be  one  of 
importance  to  deserve  double  inspection.  In  this  case,  the  area 
in  question  is  the  programs  portion  of  the  Stan/Eval  function, 
and  the  units  have  received  the  signal  clearly.  This  might 
explain  why  the  majority  of  the  units  spend  an  inordinate 
amount  of  time  preparing  for  the  programs  portion  of  the  Formal 
Visit  and  expend  relatively  little  time  preparing  for  the 
aircrew  performance  portion  of  the  visit.  The  problem  develops 
because  the  programs,  those  11  subareas,  were  instituted  to 
support  the  Stan/Eval  charter  of  measuring  the  product  quality 
of  unit  training  programs,  but  they  have  become  the  "program" 
itself.  They  are  only  the  tools  the  Stan/Eval  function  uses  to 
advise  the  commander  on  how  capable  the  aircrews  are  at 
performing  the  unit  mission;  and  as  tools,  they  are  important 
because  they  allow  unit  Stan/Eval  to  do  the  job  more 
efficiently.  But  the  goal  is  not  to  have  good  tools.  The  goal 
is  to  have  aircrews  who  can  perform  their  mission  well  because, 
in  part,  of  the  managerial  tools  that  make  up  the  Stan/Eval 
programs. 

Two  possible  ways  come  to  mind  that  could  help  redirect 
this  emphasis  on  the  programs  portion  of  the  Stan/Eval 
function.  One  method  is  to  change  the  rating  system  for  the  11 
subareas  that  make  up  the  Stan/Eval  programs.  Presently,  some 


of  them  are  rated  on  a  Sat/Unsat  scale,  some  Sat/Mar /Unsat ,  and 
the  rest  on  the  standard  -five  tier  <Unsat  to  Outstanding) 
scale.  If  the  two  tier  (Sat/Unsat)  rating  scale  was  used  for 
all  subareas,  units  would  be  less  inclined  to  expend  extra  time 
pumping  up  a  subarea  that  could  be  rated  no  higher  than  Sat. 

But  if  this  is  true,  why  do  some  units  retype  90  percent  of  the 
AF  Forms  8?  The  best  this  area  can  be  rated  is  satisfactory. 

The  reason  units  spend  this  amount  of  time  and  resources  is 
because  they  know  the  Stan/Eval  team  is  going  to  look  at  every 
one  of  the  forms  that  have  been  generated  since  the  last  Formal 
Visit,  and  the  results  are  going  to  be  distributed  to  every 
unit  in  the  command.  Consequently,  changing  the  rating  system 
will  not  change  the  emphasis  units  place  on  the  programs 
portion  of  the  Stan/Eval  function.  However,  eliminating  one  of 
the  two  redundant  inspections  does  seem  to  offer  some 
deemphasis  on  excessive  polishing  of  the  Stan/Eval  tools. 


RECOMMENDATIONS  AND  RATIONALE 


In  July  1985,  TAC/DOO  conducted  a  TAC-Gained 
Fighter/Reconnai ssance  Inspection  Review  and  concluded  that  the 
MEI  and  Formal  Visit  should  be  combined  for  TAC-gained 
units. <11:1)  For  many  of  the  same  reasons  listed  in  this  paper, 
the  conference  recommended  that,  in  essence,  one  of  the  two 
inspections  of  the  Stan/Eval  programs  portion  of  the  function 
be  dropped.  This  proposal  has  since  been  incorporated  in  the 
TACR  60-2  and  is  certainly  one  possible  solution  to  the  problem 
of  redundant  evaluations  for  all  TAC  units.  For  reasons  listed 
below,  the  recommendation  of  this  study  is  to  cut  the  Stan/Eval 
programs  portion  of  the  Formal  Visit  and  leave  the  MEI  as  is. 

In  other  words,  conduct  the  Formal  Visit  on  the  same,  existing 
frequency,  but  base  the  unit  rating  strictly  on  the  aircrew 
performance  portion  of  the  evaluation. 

1)  The  MEI  is  perfectly  capable  of  evaluating  the  programs 
portion  of  the  function.  The  IG  is  set  up  to  inspect 
management  functions.  Empirical  data  validates  their 
ability  to  reach  the  same  conclusions  as  the  Stan/Eval 
team. 

2)  The  programs  portion  of  the  function  is  no  more 
important  than  the  training,  scheduling,  or  weapons  and 
tactics  functions  and  as  such  deserves  no  more 
headquarters  scrutiny. 

3)  Eliminating  the  programs  from  the  Formal  Visit 
inspection  requirements  would  reduce  the  efforts  of 
Stan/Eval  personnel  in  performing  counterproductive  work. 
By  definition,  SEFEs  are  "the  most  highly  qualified  and 


experienced  instructor  personnel  avai lable. “ (3: 3-2) 

Freeing  them  -from  the  "massive  effort"  preparation 
requirements  of  a  Formal  Visit  makes  them  more  available 
to  per-form  instructor  duties. 

4)  Last  and  most  important,  leaving  the  evaluation  of  the 
Stan/Eval  programs  to  the  MEI  allows  the  Formal  Visit  to 
concentrate  on  evaluating  aircrew  performance — the  most 
critical  indicator  within  the  unit.  How  well  the  aircrews 
fly,  how  well  they  understand  their  weapon  system,  and  how 
well  they  know  the  threat  are  the  critical  factors.  How 
well  the  aircrews  perform  speaks  for  the  efforts  spent  in 
training,  weapons  and  tactics,  and  even  the  programs 
portion  of  Stan/Eval. 

The  type  of  aircrew  performance  seen  on  the  Formal  Visit 
is  unique.  Not  only  is  aircrew  capability  calculated  in  terms 
such  as  intercept  hack  rates,  missile  shots  or  bomb  scores  as 
the  ORI  would  measure,  it  is  also  the  result  of  how  well  the 
aircrews  planned  and  briefed  the  mission  and  the  quality  of 
flight  leadership  and  instruction  seen.  It  is  an  effective 
combination  of  measuring  mission  capability  in  an  efficient 
training  environment. 

The  Formal  Visit  is  also  unique  because  it  looks  at  the 
flight  examiners  and  instructors  as  they  perform  their  primary 
SEFE  and  instructor  duties.  These  people  are  the  core  of  the 
unit  training  programs,  and  the  quality  of  their  efforts  is  a 
key  determinant  in  the  unit’s  mission  capability.  The  Formal 
Visit  should  expand  the  flying  activities  to  include  not  only 
evaluations  for  every  SEFE  but  each  instructor  as  well.  This 
may  not  be  possible  in  training  units  where  the  number  of 
instructors  is  high,  but  the  instructor  and  SEFE  roles  are 
critical  and  deserve  this  level  of  attention  in  the  operational 
units  where  it  is  passible. 

One  last  recommendation  is  the  use  of  the  informal  visits 
by  HQ  TAC  and  NAF  SEFEs.  Fart  of  the  credit  for  the  high 
quality  of  unit  Stan/Eval  programs  across  the  command  must  go 
to  the  fact  that  double  inspection  keeps  pressure  on  the  units 
to  maintain  the  quality  of  their  programs.  Drop  one  of  the 
inspections  and  the  pressure  drops  correspond! ngl y . 

F'eriodi  cal  1  y,  these  headquarters  SEFEs  conduct  informal  visits 
in  order  to  give  required  flight  evaluations  to  unit  Chiefs  of 
Stan/Eval.  The  SEFEs  also  have  training  and  check  ride 
requirements  of  their  own.  On  these  visits,  the  SEFEs  need  to 
perform  some  of  the  standardisation  portions  of  their  duties  by 
looking  at  the  Stan/Eval  programs.  The  informal  visit 
procedures  are  adequately  described  in  TACR  60-2,  Vol .  I,  but 
with  the  programs  portion  of  the  Formal  Visit  being  deleted  as 
an  evaluation  item,  SEFEs  will  need  to  increase  the  use  of  this 
standardisation  tool  to  ensure  program  quality  remains  high. 


CONCLUSION 


Present  TAC  inspection  policy  requires  evaluation  of  the 
programs  portion  o-f  the  unit  Stan/Eval  -function  by  both  the 
Formal  Visit  and  the  MEI.  These  inspections  are  redundant,  and 
units  are  spending  an  inordinate  amount  o-f  time  preparing  -for 
them.  The  IG’ s  ability  to  rate  the  -function  has  proven  to  be 
very  accurate.  It  is  the  recommendation  o-f  this  paper  that  the 
Stan/Eval  programs  be  dropped  as  an  evaluation  item  on  the 
Formal  Visit.  The  aircrew  per-formance  portion  will  determine 
the  overall  unit  rating  and  should  be  expanded  to  include 
evaluations  -for  all  unit  SEFE  and  instructor  personnel. 
In-formal  visits  by  headquarters  SEFEs  should  include  spot 
checks  of  unit  programs  to  ensure  quality  remains  high. 
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